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This is in response to the appeal brief filed on 30 October 2006 appealing from 
the Office action mailed 06 October 2006. 

(1) Real Party in Interest 

A statement identifying by name the real party in interest is contained in the brief 
page 3. 

(2) Related Appeals and Interferences 

The examiner is not aware of any related appeals, interferences, or judicial 
proceedings which will directly affect or be directly affected by or have a bearing on the 
Board's decision in the pending appeal, [page 4] 

(3) Status of Claims 

The statement of the status of claims contained in the brief page 5 is correct. 

(4) Status of Amendments After Final 

The appellant's statement of the status of amendments after final rejection 
contained in the brief page 6 is correct. 

(5) Summary of Claimed Subject Matter 

The summary of claimed subject matter contained in the brief page 7-8 is correct. 

(6) Grounds of Rejection to be Reviewed on Appeal 

The appellant's statement of the grounds of rejection to be reviewed on appeal 
page 9 is correct. 

(7) Claims Appendix 

The copy of the appealed claims contained in the Appendix to the brief 
page 36-39 is correct. 
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(8) Evidence Relied Upon 



5,987,415 



Breese et al 



11 1999 



6,230,111 



Mizokawa 



05 2001 



(9) Grounds of Rejection 



The following ground(s) of rejection are applicable to the appealed claims: 



Claim Rejections - 35 USC § 103 



The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 1 02 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

1. Claims 1-8, 10-18 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Breese et al. [hereafter Breese], US Patent No. 5,987,415 filed on July 
30,1998, published on Nov 16, 1999 in view of Mizokawa, US Patent No. 6, 230,111 
filed on Aug 6, 1998, published on May 8, 2001 

2. As to Claims 1,8 12,18 Breese teaches a system which including 'method of 
obtaining information regarding an environment for an individual having preferred 
modalities and engaged in activity using a programmable device' [see Abstract, col 4, 
line 57-62, col 6, line 36-44,col 8, line 8-12, col 10, line 23-27, fig 3], environment for an 
individual corresponds to Breese 's computer user interface that including observing 
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user behavior, particularly user behavior caused from emotion and personality state as 
detailed in col 4, line 59-60, further it is noted that Breese specifically teaches a model 
that influence of emotion and personality based on Bayesian network as detailed in 
fig 3, programmable device corresponds to program modules being executed by a 
personal computer because program modules include processes, programs and like as 
detailed in col 6, line 40-42; 

'sensing at least one psychomotor behavioral element of the activity engaged by 
the individual' [col 8, line 23-28, line 35-41, col 10, line 34-42], Breese specifically 
teaches Bayesian net integrate various aspects of emotion and personality in a single 
model for example cognitive response as well as physical response based on the 
mental state, further, it is noted that personality nodes capture or senses individual 
emotional and or personality state as detailed in col 10, line 34-42; 

'psychomotor behavior element of the activity engaged by the individual' 
[col 12, line 49-59, fig 6], psychomotor behavior elements corresponds to loud, angry 
voice tones, calm, quite voice related to emotional state and personality of the user as 
detailed in col 12, line 50-52. 

It is however, noted that Breese does not specifically teach "determining the 
preferred modalities of the individual'. On the other hand, Mizokawa teaches a system 
which including 'determining the preferred modalities of the individual' [col 6, line 55-60], 
preferred modalities of the individual corresponds to Mizokawa's user's emotions as 
detailed in col 6, line 58-60. 
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It would have been obvious to one of the ordinary skill in the art at the time of 
applicant's invention to incorporate the teachings of Mizokawa into modeling user's 
emotion and personality in a computer user interface of Breese et al. because both 
Breese and Mizokawa are directed to user behavior and emotion model, more 
specifically Breese is directed to computer based capturing emotion and personality 
states that including multistage representation of emotional and personality variables 
[see Abstract, col 5, line 22-28], while Mizokawa is directed to controlling object using 
pseudo-emotions and pseudo-personality generated in the object, more specifically 
recognizing, updating user commands related to pseudo-personality, pseudo-emotions 
and pattern of behavior [see Abstract, fig 2-3]. 

One of the ordinary skill in the art at the time of applicant's invention to 
incorporate the teachings of Mizokawa into modeling user's emotion and personality in 
a computer user interface of Breese et al. because that would have allowed users of 
Breese to use user evaluation, emotion recognition units to observe various pattern of 
behavior or modality, further allowing to determine pseudo-emotions, predetermined 
relationship between patterns of autonomous behavior, [see Mizokawa: col 2, 
line 15-34], thus improving evaluation of user's intellectual work or desire [col 1, 
line 51-60]. 

3. As to Claim 2, Mizokawa teaches a system which including 'modifying at least 
one modifiable environmental unit to at least partially conform to the preferred 
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4. As to Claim 3, Breese teaches a system which including 'environment unit is 
modified in real-time' [col 17, line 35-43]. 

5. As to Claim 4, Mizokawa teaches a system which including 'storing the sensed 
psychomotor behavioral element in a user history' [col 7, line 32-40, line 51-55]. 

6. As to Claim 5,13,15, Breese teaches algebraic transforms' [col 2, line 38-49], 
Breese specifically teaches Bayesian network model associate with various algebraic 
transforms. On the other hand, Mizokawa teaches a system which including 'sensed 
psychomotor behavioral element is stored' [col 7, line 53-59, col 8, line 14-29], 
psychomotor behavioral element corresponds to Mizokawa's user's emotions that 
including obedience, aggressiveness, curiosity, cheerful and like as detailed in col 8, 
line 14-29. 

7. As to Claim 6, the limitations of this claim have been noted in the rejection of 
claim 1 above. In addition, Mizokawa teaches determining preferred modalities includes 
determining a preferred combination of modalities and an ordering of modalities by 
preference thereby further defining a focus of the individual's attention' [col 7, line 11- 
17, line 21-24], Mizokawa specifically teaches sound/voice recognition, analyzing 
information on the user's gestures] 
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8. As to claim 7, Mizokawa disclosed 'modifying the environmental unit to provide 
content in the environment in the preferred combination of modalities and the order of 
modalities by preference whereby the combination and the order are placed in at least 
one respective co-ordinate group of representational geometry to which attention of the 
individual is drawn, as indicated by the psychomotor behavioral element' 

[col 6, line 46-62]. 

9. As to claim 8, Breese disclosed 'defining a psychodynamic and a cognitive 
behavioral model using preferred combination modalities and the order of modalities 1 
[col 5, line 5-10, line 23-28]; 'modifying at least one environmental unit as a function of 
the psychodynamic behavioral model and the cognitive behavioral model' 

[col 8, line 35-43]. 

10. As to Claim 10, the limitations of this claim have been noted in the rejection of 
claim 1 above. And in addition, Breese teaches 'multi-dimensional and has a plurality of 
modifiable environmental units' [col 10, line 59-67]. 

11. As to claim 1 1 , Mizokawa disclosed 'preprogramming the device to monitor the 
individual for at least one specific types of psychomotor behavioral elements' [col 4, line 
25-46]; 'communicating an occurrence of the specific type of psychomotor behavioral 
element' [col 4, line 39-46]. 
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12. As to claim 14, Mizokawa disclosed 'memory device to store sensed 
psychomotor behavioral activity of the individual' [col 7, line 60-64]. 

13. As to claim 16, Mizokawa disclosed 'modalities are calculated while sensing 
psychomotor behavioral activity and concurrently used for modifications to the 
environmental units' [col 10, line 15-24]. 

14. As to claim 17, Mizokawa disclosed 'sensor includes at least one input device for 
a computer and the modifiable environmental unit includes at least one output device 1 
[fig 1,0014, line 51-59]. 

15. Claims 9 is rejected under 35 U.S.C. 103(a) as being unpatentable over 
Breese et al [hereafter Breese], US Patent No. 5,987,415 filed on Ju 30, 1998, 
published on Nov 16, 1999 in view of Mizokawa, US Patent No.6, 230,111 filed on 
Aug 6, 1998, published on May 8, 2001 

16. It is, noted both Breese, Mizokawa do not teach 'modalities calculation by an 
equation', although Breese teaches modeling a user's emotion and personality 
specifically using Bayesian network inference algorithm [see abstract, col12, line 29-35], 
while Mizokawa teaches control system for controlling object using pseudo-emotions 
and personality specifically calculating using rules or functional equations for pseudo 
emotions levels and personalities [col 10, line 15-18]. Therefore, it would have been 
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obvious to one of the ordinary skill in the art at the time of applicant's invention to not 
only sense emotions, but also establish relationship between emotions, environmental 
information to evaluate the behavioral output as suggested by Mizokawa 
[col 9, line 16-21]. 

(10) Response to Argument 

17. Applicant's arguments in the response filed on 10/30/2006, with respect to the 
rejection of claims 1-18 have been considered but not persuasive, for examiner's 
response, see discussion below: 

a) At page 15, line 6-9, claims 1,12, applicant argues that "applicant's claim of 
determining the preferred modalities of the individual is distinctly different from the 
teachings of Mizokawa's programmed responses to sensed emotions. 

b) At page 16, line 18-19, claims 1,12, "Mizokawa does not teach identifying 
preferred modalities of an individual", Mizokawa teaches recognition of emotion and a 
programmable pseudo-emotional response. Emotions are not modalities. 

As to the above argument [a-b], Mizokawa is directed to controlling object using 
pseudo-emotions and pseudo-personality generated in the object, more specifically, 
pseudo emotions, and relationship between patterns of autonomous behavior and user 
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related, and user-unrelated statuses [col 2, line 16-24], further fig 1 specifically suggests 
each element of predetermined pseudo-personality elements, pseudo-emotions and the 
and relationship. It is noted that Breese does not specifically teach, "determining the 
preferred modalities of the individual", although Breese suggests Bayesian net integrate 
various aspects of emotion and personality in a single model, and capturing or sensing 
individual emotional state [Breese: col 10, line 34-42]. 

On the other hand, Mizokawa teaches recognizing user's emotions, more 
specifically patterns of emotional expressions for example joyful, sad, surprise, angry, 
disgusted and like are captured and recognized [see Mizokawa: col 6, line 55-60], 
because, as recognized from the Mizokawa's reference that characteristics of sounds, 
voices, external information such as temperature, brightness, visually, auditory, are 
directly related to user's emotions is considered to be "modalities" of an individual 
[col 6, line 46-54], further examiner noted applicant's modalities examples in the brief 
at page 15, line 17-21, particularly examples of "modalities such as things "visual, 
auditory, kinesthetic, olfactory etc., and therefore, emotions are directly related to 
"modalities" because it senses individual emotions and causes emotional expressions 
for example joyful, sad, surprise, angry, disgusted and like are captured and recognized 
[see Mizokawa: col 6, line 55-60], further it is noted that Mizokawa specifically 
suggests relationship or mapping between emotions and characteristics of 
sounds/voices and facial expressions [col 6, line 60-67] corresponds to preferred 
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modalities of the individual. Therefore, it is sufficient to say that Mizokawa specifically 
teaches "modalities of the individual". 

c) At page 16, page 17, claim 1,12, applicant argues that "Paragraphs 66-75 
further detail modalities as claimed. As described in paragraphs 66 and 67, "when grid 
movement is determined the modality summations can be selected to determine if the 
viewer's attention is focused on visual, auditory, kinesthetic or other related cues. 
Based on the results of this equation, the web server can prepare in real time what the 
next presentation and interface should be in order to capture more of the viewer's 
attention by presenting the web content in modalities, which the viewer has 
nonconsciously selected" 

In response to the above-mention argument [c], applicant's interpretation of the 
prior art is noted. However, the claim limitation reads, "determining the preferred 
modalities of the individual". In response to applicant's argument that the references fail 
to show certain features of applicant's invention, it is noted that the features upon which 
applicant relies (i.e., "when grid movement is determined the modality summations can 

be selected to determine if the view's attention is focused on visual, auditory, ) are 

not recited in the rejected claim(s). Although the claims are interpreted in light of the 
specification, limitations from the specification are not read into the claims. 
See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993) . This is not a 
suggestion of any sort. 
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As explained in the Mizokawa's reference, Mizokawa recognizing, evaluating 
users or view's emotions, particularly several patterns of emotions that including "joyful, 
sad, surprise, angry, disgusted, and fearful [col 6, line 58-60], further user's emotions 
may also be recognized based on sound/voice and facial expressions [col 6, line 46-52], 
it is noted that Mizokawa specifically suggests relationship or mapping between 
emotions and characteristics of sounds/voices and facial expressions [col 6, line 60-67] 
corresponds to preferred modalities of the individual. 

As best understood by the examiner, claims 1 , is merely manipulation of an 
' abstract idea" without practical application or utility because it does not produce any 
"concrete", "real-world" result, although claim 1 preamble merely recites "programmable 
device", more specifically, "final result" by the claimed invention in the claim 1 elements 
particularly "determining the preferred modalities of the individual based on the 
psychomotor behavioral element of the activity engaged by the individuar do not 
produce "useful, tangible and concrete" result, The claimed invention is subject to the 
test of State Street, 149 F.3d at 1373-74, 47 USPQ2d at 1601-02. Specifically State 
Street sets forth that the claimed invention must produce a "useful, concrete result" 

Similarly, in claim 12, is merely manipulation of an " abstract idea" without 
practical application or utility because it does not produce any "concrete", "real-world" 
result, although claim 12 preamble merely recites "A programmable apparatus for 

obtaining information ", more specifically, "final result" by the claimed invention in 

the claim 1 elements particularly "a processing unit calculating the individual's 
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preferred modalities based on the sensed psychomotor behavioral activity do not 
produce "useful, tangible and concrete" result, see MPEP 2106 

Note: Examiner noted at page 18 line 5-8 [brief], "On pages 5-8 of the final office 
action, claims 2,3, 10, 13,16,17 are rejected under 35 USC 103 as allegedly being 
obvious over Rise in view of Lieberman. The limitations of claim group II are all 
directed toward modification of an environment of the individual whose activity is 
being sensed" 

In the prosecution history, claims 2,3,10,13,16,17 are rejected under 35 USC 103 
Rise in view of Lieberman in the final office action mailed on 2/14/2005, subsequently, 
examiner "re-opened" prosecution, claims 1-18 are rejected under 35 USC 103 Breese 
in view of Mizokawa, a non-final office action was mailed on 10/18/2005, therefore, 
claims 2,3,10,13,16,17 are rejected under 35 USC 103, Rise in view of Lieberman is 
NOT part of the present " Appeal Brief filed on 10/30/2006. 

d) At page 20, line 14-17, claims 1,12, applicant argues As stated in Section VII. B 
herein, neither of the references do not teach "calculating preferred modalities" 

As to the.above argument [d], examiner disagree with the applicant because 
firstly, Breese is directed to user interface including an observer capable of observing 
user behaviour [col 4, line 57-58], also Breese's network can include an observing 
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network facilitating inferencing of agent behaviour from emotion and personality states 
[col 4, line 62-66], furthermore Breese's network is a stochastic model, each stochastic 
model is preferably a Bayesian network that represents emotional and personality 
variables [col 5, line 5-6, line 11-13] for example as detailed in fig 2 

While, Mizokawa is directed to pseudo-emotions and pseudo-personality of a 
user particularly interacts with its user [see Abstract], further Mizokawa also suggests 
user's evaluation directly related to behaviour information, particularly calculating 
evaluation value from data inputted [col 6, line 38-45]. It is also noted that Mizokawa 
suggests "emotions" is correlated with the changing rate of range of the user's emotions 
for example "obedience" is correlated with frequency of the occurrence of the user's 
emotion "anger"," aggressiveness" [fig 5, col 8, line 19-35], it is also noted that level of 
each preliminary emotion can be determined by rules or functional equations correlating 
the level of each preliminary emotion with the information on the state of the user [col 9, 
line 9-13] , also emotion expression includes "changes in shapes of the eyes and the 
mouth or the shedding of tears, emotion expression can be indicated by background 
color of the "face" on the display, or by sounds or voices [col 10, line 39-45, fig 7] 
Is part of "calculating preferred modalities", as explained above, emotions are directly 
related to "modalities" because it senses individual emotions and causes emotional 
expressions for example joyful, sad, surprise, angry, disgusted and like are captured 
and recognized [see Mizokawa: col 6, line 55-60], therefore, applicant's remarks are 
deemed not to be persuasive, and claims 1-8,10-18 rejected under 35 USC 103 as 
being unpatentable over Breese in view of Mizokawa. 
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e) At page 21 , claim 2, applicant argues that "these structures do not modify the 
environment of the individual". 

As to the above argument [e], as best understood by the examiner, Mizokawa 
specifically teaches current user's state, emotion levels, user's characteristics frequency 
of occurrence of each emotion are calculated and observed how emotions have been 
changing, assigning levels for example level 2 of joy level 3 of positiveness and like as 
detailed in fig 4, col 6, line 46-54. 

f) At page 21 , claim 3, applicant argues that examiner cites to Breese for modifying 
the environment of the individual. However, the passage cited merely updates the 
network relative to information sensed from the individual. 

As to the argument [f], as best understood by the examiner, Breese specifically 
suggests Bayesian networks used to perform, inferencing from the behavioural, 
particularly related to emotion/personality states and emotional states expressed in an 
active, positive, strong, and/or terse manner given emotional/personality states [col 10, 
line 34-38], each time the state of emotional/personality change observed, the Bayesian 
network updated to reflect the changing moods of the human user as detailed in col 17, 
line 35-43]. 
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g) At page 21 , claim 10, applicant argues "it is unclear from the passage what the 
examiner regards as the dimensions of the environment in Breese and what the 
modifiable units are". 

As best understood by the examiner, multi-dimensions of the environment is 
based on the individual express themselves in an "active, positive, strong and/or terse" 
as given the emotional/personality states, further, fig 3 consisting of set of nodes 
372,374,376,378 for expression attributes part of multi-dimensions of the environment 
as detailed in col 10, line 59-67, because it related to the emotional/personality states 
which is a "multi-state nodes representing emotional and personality variables include a 
valence node, an arousal node, a friendliness node and dominance node[see col 5, 
line 23-25, col 8, line 41-60], and modifiable environmental units are part of the 
"Modeling emotions and personality", particularly Bayesian network model. 

It is also noted that the claim 10 directed to "environment is multi-dimensional", 
without defining " what is multi-dimensionaf from the specification. In the specification 
US Pub.20030074092, page 7, 0085 merely suggests, "Next, the present inventive 
method 10 can be narrowed by requiring that the environment is multi-dimensional and 
the environment has a plurality of modifiable environmental units" 

h) At page 21 , claim 1 3, Applicant argues that examiner rejects claim 1 3 on page 6 
of the final office action without identifying 1) an environmental unit of Breese; 2) an 
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instruction used to modify an environmental unit of Breese; or 3) how the environmental 
unit confirms to the preferred modality. 

As to the above argument [h], as best understood by the examiner, 
environmental unit of Breese corresponds to Modeling emotions and personality", 
particularly Bayesian network model, an instruction used to modify an environmental 
unit of Breese corresponds to conditional notation the set of probability distributions for 
a variable for example as detailed in col 2, line 54-57. On the other hand, Mizokawa 
suggests emotions are directly related to "modalities" because it senses individual 
emotions and causes emotional expressions for example joyful, sad, surprise, angry, 
disgusted and like are captured and recognized [see Mizokawa: col 6, line 55-60], col 7, 
line 53-59, col 8, line 14-29 

i) At page 24, claim 4, applicant argues "the examiner does not suggest that 
Mizokawa teaches sensing psychomotor behavioural elements. 

As to the above argument [I], as best understood by the examiner, Mizokawa 
suggests emotion expression patterns, autonomous behaviour patterns [col 7, line 46- 
47], it is also noted that Mizokawa teaches various stored units for example user stored 
in a user information in memory 1 [col 7, line 50], user's emotion recognition unit A2 , 
[col 7, line 30-31], external information recognition unit A4 getting information from 
external databases, further user's past record of using the system, various sensed 
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psychomotor behavioural elements from various sensors stored in units as detailed in 
col 7, 32-40, line 51-55. 

j) At page 25, claim 5, applicant argues "Breese does not teach storing information 
using linear algebraic transforms' 

As to the argument [j], as best understood by the examiner, Breese specifically 
teaches various algebraic transforms in computer science area, applications that 
including decision-support system, particularly in emotional signals, behavior as 
detailed in col 2, line 38-49. 

k) At page 28, claim 6, applicant argues that "nothing in Breese and Mizokawa can 
be considered to order preferred combination of modalities". 

As to the argument [k], as best understood by the examiner, Mizokawa 
specifically teaches user's emotions, and establishing relationship between emotions 
and characteristics of sounds/voices and facial expressions, further categorizing 
emotion expressions of the user into different patterns, further degree of requirement 
are recognized for example voice or speed of the user's speech and like or loudness 
corresponds to preference as detailed in col 7, line 11-17, line 21-24]. 
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I) At page 31-33, claim 9, applicant argues that specific equation for calculating a 
combination and order of modalities, neither Breese, nor Mizokawa teaches this 
equation. 

As to the above argument [I], as best understood by the examiner, Breese 
teaches Bayesian inference algorithm specifically based on user's emotion and 
personality [see Breese: Abstract, col 12, line 29-35], while Mizokawa suggests 
"pseudo-emotions" for example cheerfulness, joy and like [col 9, line 5-6], further 
Mizokawa also suggests preliminary emotion threshold is correlated with pseudo- 
personality information [col 9, line 42-53], and Mizokawa also teaches calculating the 
threshold of each preliminary emotion are compared with various levels for example 
higher than the preliminary emotion thresholds or lower than the preliminary emotions , 
although it is noted that both Breese, Mizokawa do not show "modalities calculation by 
an equation". 

Therefore, it would have been obvious to one of the ordinary skill in the art at the 
time of applicant's invention to combine the Breese, and Mizokawa's references to 
N derive modalities calculation by an equation because Breese directed to stochastic 
models [algorithms] of various levels of emotions/personalities that represents 
"human behavior", while Mizokawa teaches calculating, comparing pseudo-emotions 
in the "pseudo-personality" thus, establishing relationship between emotins, 
environmental information to evaluate the behavioral output as suggested by 
Mizokawa [col 9, line 16-21]. 
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(1 1 ) Related Proceeding(s) Appendix 

No decision rendered by a court or the Board is identified by the examiner in the 
Related Appeals and Interferences section of this examiner's answer. 



For the above reasons, it is believed that the rejections should be sustained. 
Respectfully submitted, 

Srirama Channavajjala 



Conferees: 




Alam, Hosain 
SPE.AU2166 
November 30, 2006 



Lee Eddie.G' 
SPE.TC2100 




